Maintainable Dynamic Instrumentation Technique For Changing Versions Of Software

ABSTRACT

A technique for the dynamic instrumentation of a running software system. One or more callable instrumentation functions are accessible in a first memory space associated with the software system. The one or more callable instrumentation functions are adapted to probe an operation of the software system and return data regarding the probed operation. Probed operation environment information needed by the one or more instrumentation functions is provided to a second memory space associated with the software system. First memory space addresses associated with the probed operation environment information are determined from a resource that is accessible in the second memory space. A probe handler is generated that includes calls to the one or more instrumentation functions with references to the first memory space addresses. The probe handler is callable as part of the probed operation.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to diagnostic tools for analyzingsoftware. More particularly, the invention is directed to the dynamicinstrumentation of running software in order to gather runtime data asthe software executes. Still more particularly, the invention concerns atechnique for implementing a portable dynamic instrumentation tool thataccommodates variations in execution environment that arise fromdifferences in compiler versions, compile options, hardware platformparticulars and other parameters.

2. Description of the Prior Art

By way of background, in order to analyze the operation of a complexsystem such as a large software program, it is important to be able togather operational information while the system is running, withouthalting execution, modifying the system, and then restarting it. Oneapproach is to dynamically add instrumentation code to the runningsystem when data is needed, so that there is no performance penalty.This is in contrast to statically instrumenting the system when it iscreated, which offers no flexibility. Operating systems are examples ofcomplex systems that may require diagnosis at run time. As shown in FIG.1A, it is often desirable to probe some kernel function to determineinformation about its operation, such as how often or why the functionis being called. The kernel memory location where the function is to beprobed is referred to as the probe point. As shown in FIG. 1B, dynamicinstrumentation involves adding a software interrupt at the probe pointin the running system and providing a probe handler that is executedwhen the interrupt throws an exception. The probe handler executesinstrumentation code (instrumentation function) that obtains therequired information and provides it to the probe user, then returnscontrol to the running system. Examples of such conventional dynamicinstrumentation software include the Linux® System Tap framework. Thisframework provides a Linux® tool that allows application developers andsystem administrators to write probe scripts that will be automaticallyparsed, elaborated upon, translated, compiled and loaded as a kernelmodule to generate probe data from a running kernel.

One problem with conventional instrumentation solutions is that theprobe handler needs the same execution environment as that of thefunction being interrrupted, including access to local variables andarguments. Unfortunately, the location of local variables and argumentsis not fixed, and depends on various parameters such as the compilerversion used to compile the operating system, and the compile optionsand hardware platform type specified during compilation. For thisreason, local variables and arguments can not be easily accessed in theprobe handler without consulting additional debug information. It is notdesirable to access debug information from a probe handler insofar asthis information is generally not readily available to the operatingsystem. For example, the kernel debug information is typicallymaintained in a file system that is readiliy accessible in user spacebut not from kernel context. Due to this problem, it is not easy towrite dynamic instrumentation code in a portable form. The only knownalternative is to write the probe handler code for a specific version ofthe compiler, compile options and platform being utilized. It will beappreciated that this method is not desirable insofar as it is notpractical or economical to write such code.

Accordingly, it would be desirable to provide a techinque whereby aprobe handler could reference the same execution environment as that ofthe function being interrupted without consulting debug information fromthe handler. The technique should be powerful enough so that it can beextended to dynamically add new functionality or fix a problem in anexisting system without taking the system down.

SUMMARY OF THE INVENTION

The foregoing problems are solved and an advance in the art is obtainedby a novel method, system and computer program product for the dynamicinstrumentation of a running software system. One or more callableinstrumentation functions are accessible in a first memory spaceassociated with the software system. The one or more callableinstrumentation functions are adapted to probe an operation of thesoftware system and return data regarding the probed operation. Probedoperation environment information needed by the one or moreinstrumentation functions is provided to a second memory spaceassociated with the software system. First memory space addressesassociated with the probed operation environment information aredetermined from a resource that is accessible in the second memoryspace. A probe handler is generated that includes calls to the one ormore instrumentation functions with references to the first memory spaceaddresses. The probe handler is callable as part of the probedoperation.

According to exemplary embodiments disclosed herein, the software systemmay be an operating system kernel, the first memory space may be aprivileged memory space in which the operating system kernel resides,and the second memory space may be a non-privileged memory space inwhich user applications reside. The one or more callable instrumentationfunctions may be placed in the first memory space in conjunction withthe software system being placed in the first memory space. The probedoperation environment information may comprise function arguments andlocal variables, and may be provided to the second memory space by wayof a registration interface in the first memory space that allows theone or more callable instrumentation functions to register the probedoperation environment information. A database management system may alsobe provided in the first memory space that associates the one or moreinstrumentation functions and the registered probed operationenvironment information with the probed operation. The first memoryspace may further include communication logic that receives the probedoperation environment information from the database management systemand provides it to the second memory space. The first memory spaceaddresses may be determined from a debug information resource in a filesystem that is accessible from the second memory space. The probehandler may be generated based on a probe script provided in the secondmemory space.

In another aspect, the invention may be thought of as providing supportfor dynamic instrumentation of a running software system by establishingbuild-independent core instrumentation logic and build-dependent dynamicinstrumentation logic. The core instrumentation logic is adapted toprobe an operation of the software system and obtain data regarding theprobed operation. The core instrumentation logic is independent of buildparameters used to compile the software system. The dynamicinstrumentation logic provides build-dependent probed operationenvironment information to the core instrumentation logic. The probedoperation environment information is dependent on build parameters usedto compile the software system. The core instrumentation logic isadapted to use the build-dependent probed operation environmentinformation to probe an operation of the software system. A firstinterface may be provided that enables the core instrumentation logic toprovide a specification of build-independent probed operationenvironment information to a dynamic instrumentation logic generator. Asecond interface may be provided that allows the dynamic instrumentationlogic to provide build-dependent probed operation environmentinformation to the core instrumentation logic when the latter isinvoked.

In a further aspect, the invention may be thought of as providingsupport for dynamic instrumentation of a running software system byestablishing an architecture adapted to create dynamic instrumentationcode for probing an operation of the software system and exporting dataregarding the probed operation. The architecture is adapted to generatea specification of build-independent probed operation environmentinformation that is required to instrument the software system. Thearchitecture is further adapted to consult an information resourceassociated with the software system in order to obtain build-dependentprobed operation environment information based on the specification thatis dependent on build parameters used to compile the software system.The architecture is still further adapted to generate a dynamicinstrumentation module that uses the build-dependent probed operationenvironment information to probe the software system. The architecturemay be implemented with a core instrumentation logic module, a dynamicinstrumentation logic generator and a dynamic instrumentation module.The core instrumentation module is adapted to probe an operation of thesoftware system using the build-dependent probed operation environmentinformation and to obtain data regarding the probed operation. Thedynamic instrumentation logic generator creates the dynamicinstrumentation module. The core instrumentation logic further providesthe specification of the build-independent probed operation environmentinformation to the dynamic instrumentation logic generator. The dynamicinstrumentation logic generator uses the specification to consult theinformation resource associated with the software system and obtain thebuild-dependent probed operation environment information used to createthe dynamic instrumentation logic. The dynamic instrumentation moduleprovides the build-dependent probed operation environment information tothe core instrumentation logic module.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features and advantages of the invention will beapparent from the following more particular description of exemplaryembodiments of the invention, as illustrated in the accompanyingDrawings, in which:

FIG. 1A is a diagrammatic illustration of a portion of a kernel memoryspace containing operating system kernel code, including a function ofinterest to be probed using a prior art technique;

FIG. 1B is a diagrammatic illustration of the kernel memory spaceportion of FIG. 1A showing the insertion of a software interrupt at aprobe point associated with the function of interest and further showinganother kernel memory space portion containing a probe handler modulethat executes an instrumentation function according to a prior arttechnique;

FIG. 2 is a is a functional block diagram showing components of adynamic instrumentation system adapted to perform dynamicinstrumentation of running software in accordance with the invention;

FIG. 3 is a diagrammatic illustration of a database management systemmaintained by a central module of the dynamic instrumentation system ofFIG. 2;

FIG. 4 is a flow diagram showing dynamic instrumentation functionsperformed by the dynamic instrumentation system of FIG. 2;

FIG. 5 is a block diagram showing an exemplary hardware environment thatmay be used to host the dynamic instrumentation system of FIG. 2; and

FIG. 6 is a diagrammatic illustration showing physical media that may beused to provide a computer program product for implementing a dynamicinstrumentation system in accordance with the invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Turning now to the Drawing figures wherein like reference numeralsindicate like components in all of the several views, FIG. 2 illustratesan exemplary dynamic instrumentation system 2. Although theinstrumentation system 2 may be variously implemented in software,firmware, hardware, or any combination thereof, the following discussionis based on an exemplary embodiment wherein the instrumentation system 2comprises software program logic that is capable of execution oncommodity data processing systems, including personal computers,workstations, as well as more complex hardware such as midrange andmainframe computers. According to this exemplary softwareimplementation, the software logic of the instrumentation system 2 maycollectively reside in a first memory space 2A (shown on the left-handside of FIG. 2) and a second memory space 2B (shown on the right-handside of FIG. 2), so as to be available for execution on a centralprocessing unit (CPU) (see discussion of FIG. 5 below). A softwaresystem 4, representing a dynamic instrumentation target, additionallyresides in the first memory space 2A. To further facilitate the presentdiscussion, it will be assumed by way of example only that the softwaresystem 4 represents a base operating system, such as the Linux® kernel,and that the first memory space 2A comprises an operating system kernelmemory (e.g., a privileged memory space) in which the software system 4resides, and a second memory space 2B that may be a user applicationmemory space (e.g., a non-privileged memory space) in which resides userapplications. It should be understood that the disclosed embodimentrepresents only one exemplary implementation of the invention, and thatthe dynamic instrumentation system 2 could also be used with other typesof software system 4.

The instrumentation system 2 is adapted to provide dynamicinstrumentation of the software system 4 to facilitate runtime diagnosisthereof. As described by way of background above, dynamicinstrumentation refers to the ability to dynamically insertinstrumentation code in a running system, as opposed to staticallyinserting the instrumentation code at compile time. The dynamicinstrumentation system 2 provides such dynamic instrumentationcapability while offering the additional capability of accessing thesame execution environment (e.g. local variables and arguments) as thesoftware system 4 without regard to how the software system 4 wascompiled, the compiler options that were used, or the hardware platformtype specified during compilation. To that end, the instrumentationsystem 2 includes one or more core instrumentation functions 6 ₁, 6 ₂ .. . 6 _(n) representing probe logic that is accessible within the firstmemory space 2A. According to the present embodiment in which thesoftware system 4 is exemplified as an operating system kernel, theinstrumentation functions 6 ₁, 6 ₂ . . . 6 _(n) may be loaded asexecutable code into the first memory space 2A in conjunction with thekernel (software system 4) as a kernel module. This kernel module isshown in FIG. 2 as an instrumentation library module 6. The one or moreinstrumentation functions 6 ₁, 6 ₂ . . . 6 _(n) are adapted to probeoperations of the software system 4 and export data regarding the probedoperations. The instrumentation functions are considered to bebuild-independent because they are not hard-coded with the memoryaddresses of the arguments and local variables of the software systemoperations they are designed to probe, which are build-dependent.

Each instrumentation function 6 ₁, 6 ₂ . . . 6 _(n) is designed to probea particular operation of the software system 4, such as a kernelfunction if the software system 4 is an operating system kernel. Forexample, if the software system 4 is a Linux® kernel, theinstrumentation function 61 might be designed to probe the Linux® kernelfunction known as sys_read( ), which is a service function for the read() system call used to read data from a file. Similarly, theinstrumentation function 62 might be designed to probe the Linux® kernelfunction known as sys_write( ), which is a service function for thewrite( ) system call used to write data to a file. The otherinstrumentation functions 6 ₃-6 _(n) may be designed to probe otherLinux kernel functions. One purpose of the instrumentation functions 6 ₁and 6 ₂ might be to generate a tally of the frequency at which thesys_read( ) or sys_write( ) system calls are invoked, the files that areinvolved, the number of bytes being read and written, and whatapplications are accessing the files.

The Linux® sys read( ) and sys write( ) functions take the arguments“fd,” “buf”and “count,” where “fd” is the file descriptor of the file tobe accessed, “buf” is the memory buffer address for the data to betransferred, and “count” is the number of bytes to be transferred. TheLinux® sys read( ) and sys write( ) functions use the internal variable“ret” to track the number of bytes transferred (and return this value tothe caller). The foregoing function arguments and internal variables ofthe sys_read( ) and sys_write( ) functions may be referred to as probedoperation environment information. It is assumed that there will be someinstrumentation function (e.g., 6 ₁ or 6 ₂) that operates on suchenvironment information when probing the function in question.

Because the instrumentation functions 6 ₁, 6 ₂ . . . 6 _(n) do not knowthe build-dependent addresses of their probed operation environmentinformation in advance, the instrumentation functions 6 ₁, 6 ₂ . . . 6_(n) may be written as callable functions that can be passed therequired information as arguments when the instrumentation functions areinvoked. As described in more detail below, a dynamic probe handlermodule 8 is loaded into the first memory space 2A (as executable code)for this purpose. Three basic tasks of the probe handler module 8 are toallocate local variables, call one or more of the instrumentationfunctions 6 ₁, 6 ₂ . . . 6 _(n), and perform routine memory clean upoperations when probe execution has terminated. The probe handler module8 is also responsible for exporting probe results to the second memoryspace 2B for user consumption. Alternatively, the instrumentationfunctions 6 ₁, 6 ₂ . . . 6 _(n) could be designed to export theirresults directly.

Before the probe handler module 8 can be generated and loaded into thefirst memory space 2A, a technique is needed for determining thebuild-dependent memory addresses of the probed operation environmentinformation for use by the instrumentation functions 6 ₁, 6 ₂ . . . 6_(n). An exemplary technique used by the instrumentation system 2 is toprovide a specification of the probed operation environment informationneeded by the instrumentation functions 6 ₁, 6 ₂ . . . 6 _(n) to thesecond memory space 2B associated with the software system 4. Using thisspecification, the required address information can be obtained in thesecond memory space 2B and used to generate the probe handler 8.

A central module 10 accessible within the first memory space 2A allowsall probed operation environment information used by the instrumentationfunctions 6 ₁, 6 ₂ . . . 6 _(n) to be specified to the second memoryspace 2B. The central module 10 may include an instrumentation functionregistration API (application program interface) 12, a probepoint/instrumentation function database management system (DBMS) 14, anda user communication daemon 16. The instrumentation function API 12provides an interface that the instrumentation functions 6 ₁, 6 ₂ . . .6 _(n) may use to register their probed operation environmentinformation with the central module 10. This registration may beperformed when the instrumentation library module 6 is first activated,and whenever new instrumentation functions are added. The registrationinformation may include the name of the instrumentation function 6 ₁, 6₂ . . . 6 _(n), the operation in the software system 4 to be probed bythe instrumentation function, and the probed operation environmentinformation.

The DBMS 14 maintains a database that stores the registrationinformation registered via the instrumentation function API 12. Itassociates the one or more instrumentation functions 6 ₁, 6 ₂ . . . 6_(n) and their probed operation environment information with the probedoperations. For example, as shown in FIG. 3, and if the software system4 is a Linux® kernel, an instrumentation function known as“probe_on_sys_read( )” (e.g., instrumentation function 6 ₁ in FIG. 2) isassociated with the sys_read( ) kernel function as the probe point,together with the function arguments and local variables of thesys_read( ) function as the probed operation environment information.Similarly, an instrumentation function known as “probe_on_sys_write( )”(e.g., instrumentation function 6 ₂ in FIG. 2) is associated with thesys_write( ) kernel function as the probe point, together with thefunction arguments and local variables of the sys_write( ) function asthe probed operation environment information.

The user communication daemon 16 represents executable code thatprovides a communication mechanism for passing the probed operationenvironment information stored in the DBMS 14 to a probe handlergenerator 18 associated with the second memory space 2B. Thecommunication daemon 16 may implement a “push” mechanism that passes thecontents of the DBMS 14 to the probe handler generator 18 whenever thereis a change in the DBMS (e.g. when an instrumentation function registerswith the registration API 12). This information may be stored in anysuitable fashion in the user space 2B, such as in a memory buffer or adisk file accessible to the probe handler generator 18. Alternatively,the communication daemon 16 could implement a “pull” mechanism wherebythe contents of the DBMS 14 are only provided to the probe handlergenerator 18 upon the latter's request.

The probed operation environment information received from thecommunication daemon 16 is build-independent because it does notcomprise actual memory addresses in the first memory space 2A, onlyargument and internal variable names. In order to generatebuild-dependent probed operation environment information, the probehandler generator 18 obtains the (first memory space) addressesassociated with the probed operation environment information from aresource that is accessible in the second memory space 2B. A debuginformation resource stored in a file system that is accessible from thesecond memory space 2B may be used for this purpose. For example, asshown in FIG. 2, if the software system 4 is an operating system kernel,this resource could be a conventional file or directory that maintainskernel debug information 22 (e.g., the Linux® System.map symbol table).The kernel debug information 22 represents information that can begenerated and preserved during conventional compilation of an operatingsystem kernel. Similar resources may be created during the compilationof other types of the software system 4, using known techniques. Thus,using the build-independent information obtained from the communicationdaemon 16, the probe handler generator 18 can obtain the build-dependent(first address space) memory address of all probed operation environmentinformation required by the instrumentation function(s) associated withthe probed operation.

Having obtained the first memory space addresses for the probedoperation environment information, the probe handler generator 18constructs the probe handler module 8 according to the user probe script20. In this regard, the probe handler generator 18 may be implementedusing conventional instrumentation techniques such as those used in theopen source Linux® System Tap framework described by way of backgroundabove. Such instrumentation techniques have applicability not only tothe present embodiment wherein the software system 4 is exemplified asan operating system kernel, but to many other types of software system4, such as application programs, runtime libraries, etc. Once the probehandler 8 is generated in the second memory space 2B, the probe handlergenerator 18 loads it into the first memory space 2A for operation asbuild-dependent dynamic instrumentation logic.

As described above, the probe handler module 8 includes calls to one ormore of the instrumentation functions 6 ₁, 6 ₂ . . . 6 _(n). Thesefunction calls pass references (e.g., pointers) to the first memoryspace addresses for the probed operation environment informationobtained from a second memory space resource such as the kernel debuginformation 22. The core build-independent logic of the instrumentationfunctions 6 ₁, 6 ₂ . . . 6 _(n) may thereby be invoked using thebuild-dependent probed operation environment information obtained fromthe second memory space 2B. Advantageously, the build-independent logicof the instrumentation functions 6 ₁, 6 ₂ . . . 6 _(n) can be usedwithout modification in a large number of environments that varyaccording to compiler selection, compile options and hardware platform.Using the dynamic instrumentation technique described above, allbuild-dependent probed operation environment information can be readilyobtained and utilized by the core instrumentation function logic.

Turning now to FIG. 4, the operations performed by the various logicentities of the instrumentation system 2 may be further understood whenviewed within the context of a flow diagram. The following assumes thatthe instrumentation module 6 and the central module 10 have been loadedin the first memory space 2A, and that the probe handler generator 18 isoperational in the second memory space 2B. In step 30 of FIG. 4, theinstrumentation functions 6 ₁, 6 ₂ . . . 6 _(n) register with thecentral module 10 via the instrumentation function registration API 12.In step 32, the instrumentation function API 12 populates the probepoint/instrumentation function DBMS 14 based on the registrationinformation provided in step 30. In step 34, the DBMS 14 pushes newdatabase entries to the communication daemon 16. In step 36, thecommunication daemon 16 pushes the new database entries to the userspace 2B. As described above, each entry will identify aninstrumentation function, along with the associated probe point andprobed operation environment information.

In step 38, a user probe script 20 is submitted to the probe handlergenerator 18. The probe handler generator 18 parses the user probescript 20 and determines the operation(s) of the software system 4 to beprobed. In step 40, the probe handler generator 18 retrieves thedatabase information from the location in the user space 2B where it wasplaced by the communication daemon 16 (e.g., a buffer, a file, etc.). Instep 42, the probe handler generator 18 consults the resource in thesecond memory space 2B that maintains the first memory addresses for theprobed operation environment information. As an example, if the softwaresystem 4 is an operating system kernel, this could be the file ordirectory resource that stores the kernel debug information 22. In step44, the probe handler generator 18 constructs the probe handler module 8(e.g. as a compiled kernel module). As described above, the probehandler module 8 is designed to call all instrumentation functions 6 ₁,6 ₂ . . . 6 _(n) corresponding to probed operations specified in theuser probe script 20, and to pass references to the first memory spaceaddresses determined in the second memory space resource (e.g., thekernel debug information 22) as parameters.

In step 46, the probe handler generator 18 loads the probe handlermodule 8 in the first memory space 2A. The module loading procedure,which is conventional in nature, will set a software interrupt in thesoftware system 4 at the desired probe point(s). When the probe handlermodule 8 is invoked by the software interrupt, it calls one of theinstrumentation functions 6 ₁, 6 ₂ . . . 6 _(n) in step 48. The calledinstrumentation function 6 ₁, 6 ₂ . . . 6 _(n) returns from processingto the probe handler module 8 in step 50. If the probe handler module 8is adapted to export probe return data, the return from the calledinstrumentation function 6 ₁, 6 ₂ . . . 6 _(n) will include this data.The probe return data can then be exported to the second memory space 2Bin step 52.

Accordingly, a technique for dynamic instrumentation of running softwarehas been disclosed. It will be appreciated that the foregoing conceptsmay be variously embodied in any of a data processing system, a machineimplemented method, and a computer program product in which programminglogic is provided by one or more machine-readable media for use incontrolling a data processing system to perform the required functions.Relative to a data processing system and machine implemented method,FIG. 5 illustrates an exemplary hardware environment 60 is shown inwhich the instrumentation system 2 may be implemented. The hardwareenvironment 60 includes a CPU or other data processing resource 62, aphysical memory 64 providing the first memory space 2A and the secondmemory space 2B, a graphics card 66 for generating visual output to amonitor (not shown), a peripheral storage device 68, other peripheraldevices 70, and a bus infrastructure 72 interconnecting the foregoingelements. The instrumentation library 6, the probe handler module 8, thecentral module 10 and its API, DBMS, and communication components 12, 14and 16, and the probe handler generator 18 may all be loaded in thephysical memory 64. The kernel debug information 22 or other resourcestoring first memory space addresses for probed operation environmentinformation may reside on the storage device 68. The user probe script20 may be input through user interaction with peripheral devices 70(e.g., keyboard, mouse, etc.).

Relative to a computer program product having a machine-readable mediaand programming logic for controlling a data processing system,exemplary machine-readable media for providing such programming logicare shown by reference numeral 100 in FIG. 6. The media 100 are shown asbeing portable optical storage disks of the type that are conventionallyused for commercial software sales, such as compact disk-read onlymemory (CD-ROM) disks, compact disk-read/write (CD-R/W) disks, anddigital versatile disks (DVDs). Such media can store the programminglogic of the invention, either alone or in conjunction with anothersoftware product that incorporates the required functionality. Theprogramming logic could also be provided by portable magnetic media(such as floppy disks, flash memory sticks, etc.), or magnetic mediacombined with drive systems (e.g. disk drives), or media incorporated indata processing platforms, such as random access memory (RAM), read-onlymemory (ROM) or other semiconductor or solid state memory. More broadly,the media could comprise any electronic, magnetic, optical,electromagnetic, infrared, semiconductor system or apparatus or device,transmission or propagation signal or signal-carrying medium (such as anetwork), or other entity that can contain, store, communicate,propagate or transport the programming logic for use by or in connectionwith a data processing system, computer or other instruction executionsystem, apparatus or device.

Although various embodiments of the invention have been described, itshould be apparent that many variations and alternative embodimentscould be implemented in accordance with the invention. It is understood,therefore, that the invention is not to be in any way limited except inaccordance with the spirit of the appended claims and their equivalents.

1. A method for the dynamic instrumentation of a running softwaresystem, comprising: providing one or more callable instrumentationfunctions that are accessible in a first memory space associated withsaid software system; said one or more callable instrumentationfunctions being adapted to probe an operation of said software systemand return data regarding said probed operation; providing probedoperation environment information needed by said one or moreinstrumentation functions to a second memory space associated with saidsoftware system; determining first memory space addresses associatedwith said probed operation environment information from a resource thatis accessible in said second memory space; and generating a probehandler that includes calls to said one or more instrumentationfunctions with references to said first memory space addresses, saidprobe handler being callable as part of said probed operation.
 2. Amethod in accordance with claim 1 wherein said software system is anoperating system kernel, said first memory space is a privileged memoryspace in which said operating system kernel resides, and said secondmemory space is a non-privileged memory space in which user applicationsreside.
 3. A method in accordance with claim 1 wherein said one or morecallable instrumentation functions are placed in said first memory spacein conjunction with said software system being placed in said firstmemory space.
 4. A method in accordance with claim 1 wherein said probedoperation environment information comprises function arguments and localvariables.
 5. A method in accordance with claim 1 wherein said probedoperation environment information is provided to said second memoryspace by way of a registration interface in said first memory space thatallows said one or more callable instrumentation functions to registersaid probed operation environment information.
 6. A method in accordancewith claim 5 wherein said probed operation environment information isfurther provided to said second memory space by way of a databasemanagement system in said first memory space that associates said one ormore instrumentation functions and said registered probed operationenvironment information.
 7. A method in accordance with claim 6 whereinsaid probed operation environment information is further provided tosaid second memory space by way of said database management systemproviding said probed operation environment information to communicationlogic in said first memory space in response to an update to saiddatabase management system.
 8. A method in accordance with claim 7wherein said probed operation environment information is furtherprovided to said second memory space by way of said communication logicproviding said probed operation environment information received fromsaid database management system to said second memory space.
 9. A methodin accordance with claim 1 wherein said first memory space addresses aredetermined from a debug information resource in a file system that isaccessible from said second memory space.
 10. A method in accordancewith claim 1 wherein said probe handler is generated based on a probescript provided in said second memory space.
 11. A data processingsystem programmed for dynamic instrumentation of a running softwaresystem, comprising: a data processing resource; a first memory space; asecond memory space; one or more callable instrumentation functionsaccessible in said first memory space; said one or more callableinstrumentation functions being adapted to probe an operation of saidsoftware system and return data regarding said probed operation; firstlogic in said first memory space adapted to provide probed operationenvironment information needed by said one or more instrumentationfunctions to said second memory space; and second logic in said secondmemory space adapted to adapted to determine first memory spaceaddresses associated with said probed operation environment informationand generate a probe handler that includes logic for making calls tosaid one or more instrumentation functions with references to said firstmemory space addresses.
 12. A data processing system in accordance withclaim 11 wherein said software system is an operating system kernel,said first memory space is a privileged memory space in which saidoperating system kernel resides, and said second memory space is anon-privileged memory space in which user applications reside.
 13. Adata processing system in accordance with claim 11 wherein said one ormore callable instrumentation functions reside in said first memoryspace in conjunction with said software system.
 14. A data processingsystem in accordance with claim 11 wherein said probed operationenvironment information comprises function arguments and localvariables.
 15. A data processing system in accordance with claim 11further including a registration interface in said first memory spacethat allows said one or more callable instrumentation functions toregister said probed operation environment information.
 16. A dataprocessing system in accordance with claim 15 further including adatabase management system in said first memory space that associatessaid one or more instrumentation functions and said registered probedoperation environment information with said probed operation.
 17. A dataprocessing system in accordance with claim 16 wherein said first logicin said first memory space comprises communication logic that providessaid probed operation environment information to said second memoryspace.
 18. A data processing system in accordance with claim 17 whereinsaid communication logic is further adapted to receive said probedoperation information from said database management system when saiddatabase management system is updated.
 19. A data processing system inaccordance with claim 11 wherein said first memory space addresses aredetermined from a debug information resource in a file system that isaccessible from said second memory space.
 20. A data processing systemin accordance with claim 11 wherein said second logic in said secondmemory space is adapted to generate said probe handler based on a probescript provided in said second memory space.
 21. A computer programproduct for the dynamic instrumentation of a running software system,comprising: one or more machine readable media; programming logicprovided by said machine readable media for programming a dataprocessing platform to support dynamic instrumentation of said runningsoftware system, as by: providing one or more callable instrumentationfunctions that are accessible in a first memory space associated withsaid software system; said one or more callable instrumentationfunctions being adapted to probe an operation of said software systemand return data regarding said probed operation; providing probedoperation environment information needed by said one or moreinstrumentation functions to a second memory space associated with saidsoftware system; determining first memory space addresses associatedsaid probed operation environment information from a resource that isaccessible in said second memory space; and generating a probe handlerthat includes calls to said one or more instrumentation functions withreferences to said first memory space addresses, said probe handlerbeing callable as part of said probed operation.
 22. A computer programproduct in accordance with claim 21 wherein said software system is anoperating system kernel, said first memory space is a privileged memoryspace in which said operating system kernel resides, and said secondmemory space is a non-privileged memory space in which user applicationsreside.
 23. A computer program product in accordance with claim 21wherein said programming logic is further adapted to place said one ormore callable instrumentation functions in said first memory space inconjunction with said software system being placed in said first memoryspace.
 24. A computer program product in accordance with claim 21wherein said probed operation environment information comprises functionarguments and local variables.
 25. A computer program product inaccordance with claim 21 wherein said programming logic is furtheradapted to provide a registration interface in said first memory spacethat allows said one or more callable instrumentation functions toregister said probed operation environment information.
 26. A computerprogram product in accordance with claim 25 wherein said programminglogic is further adapted to provide a database management system in saidfirst memory space that associates said one or more instrumentationfunctions and said registered probed operation environment informationwith said probed operation.
 27. A computer program product in accordancewith claim 26 wherein said programming logic is further adapted toprovide communication logic in said first memory space that providessaid probed operation environment information to said second memoryspace.
 28. A computer program product in accordance with claim 27wherein said communication logic is further adapted to receive saidprobed operation environment information from said database managementsystem when said database management system is updated.
 29. A computerprogram product in accordance with claim 21 wherein said first memoryspace addresses are determined from a debug information resource in afile system that is accessible from said second memory space.
 30. Acomputer program product in accordance with claim 21 wherein said probehandler is generated based on a probe script provided in said secondmemory space.
 31. A computer program product for the dynamicinstrumentation of a running software system, comprising: one or moremachine readable media; programming logic provided by said machinereadable media for programming a data processing platform to supportdynamic instrumentation of said running software system, as by:establishing core instrumentation logic adapted to probe an operation ofsaid software system and obtain data regarding said probed operation;said core instrumentation logic being independent of build parametersused to compile said software system; establishing dynamicinstrumentation logic that provides build-dependent probed operationenvironment information to said core instrumentation logic; said probedoperation environment information being dependent on build parametersused to compile said software system; and said core instrumentationlogic being adapted to use said build-dependent probed operationenvironment information to probe an operation of said software system.32. A computer program product in accordance with claim 31 wherein saidprogramming logic is further adapted to provide a first interface thatallows said core instrumentation logic to provide a specification ofbuild-independent probed operation environment information to a dynamicinstrumentation logic generator.
 33. A computer program product inaccordance with claim 32 wherein said programming logic is furtheradapted to provide a second interface that allows said dynamicinstrumentation logic to provide build-dependent probed operationenvironment information based on said specification to said coreinstrumentation logic when said core instrumentation logic is invoked.34. A computer program product for the dynamic instrumentation of arunning software system, comprising: one or more machine readable media;programming logic provided by said machine readable media forprogramming a data processing platform to support dynamicinstrumentation of said running software system, as by: establishing anarchitecture adapted to create dynamic instrumentation code for probingan operation of said software system and exporting data regarding saidprobed operation; said architecture being adapted to generate aspecification of build-independent probed operation environmentinformation that is required to instrument said software system; saidarchitecture being further adapted to consult an information resourceassociated with said software system in order to obtain build-dependentprobed operation environment information based on said specificationthat is dependent on build parameters used to compile said softwaresystem; and said architecture being adapted to generate a dynamicinstrumentation module that uses said build-dependent probed operationenvironment information to probe said software system.
 35. A computerprogram product in accordance with claim 34 wherein said architecturecomprises: a core instrumentation logic module adapted to probe anoperation of said software system using said build-dependent probedoperation environment information and obtain data regarding said probedoperation; a dynamic instrumentation logic generator adapted to generatesaid dynamic instrumentation module; said core instrumentation logicbeing further adapted to provide said specification of saidbuild-independent probed operation environment information that itrequires to said dynamic instrumentation logic generator; saidinstrumentation logic generator being further adapted to use saidspecification to consult said information resource associated with saidsoftware system and obtain said build-dependent probed operationenvironment information; and said dynamic instrumentation module beingadapted to provide said build-dependent probed operation environmentinformation to said core instrumentation logic module.